COMMUTATION RELATIONS AND MARKOV CHAINS 



JASON FULMAN 

Abstract. It is shown that the combinatorics of commutation rela- 
tions is well suited for analyzing the convergence rate of certain Markov 
chains. Examples studied include random walk on irreducible represen- 
tations, a local random walk on partitions whose stationary distribution 
is the Ewens distribution, and some birth-death chains. 



1. Introduction 

Stanley |St2j introduced a class of partially ordered sets, which he called 
differential posets, with many remarkable combinatorial and algebraic prop- 
erties. A basic tool in his theory was the use of two linear transformations 
U and D on the vector space of linear combinations of elements of P. If 
X € P then Ux (respectively, Dx) is the sum of all elements covering x (re- 
spectively, which X covers). For differential posets one has the commutation 
relation DU — UD = rl for some positive integer r, and he exploited this 
to compute the spectrum and eigenspaces (though typically not individual 
eigenvectors) of the operator UD. 

The primary purpose of this paper is to show that commutation relations 
are useful not only for studying spectral properties, but also for obtaining 
sharp Markov chain convergence rate results. We will need the more general 
commutation relation (studied in Fomin's paper [Fo| ) 

(1-1) Dn+lUn = anUn-lDn + bnJn, 

for all n. In many of our examples the operators U, D will not be Stanley's 
up and down operators but will be probabilistic in nature and will involve 
certain weights. 

There are several ways of quantifying the convergence rate of a Markov 
chain K to its stationary distribution vr. These, together with other proba- 
bilistic essentials, will be discussed in Section [2l For now we mention that 
the commutation relations (II. Ih will be particularly useful for studying the 
maximal separation distance after r steps, defined as 

K-ix,yy 



s*{r) := max 



1 



7r(y) 
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where K^{x,y) is the chance of transitioning from x to y in r steps. In 
general it can be quite a subtle problem even to determine which x, y attain 
the maximum in the definition of s*(r). Our solution to this problem involves 
using the commutation relations (jl.ip to write ^J^f^ as a sum of non- 
negative terms. 

After determining which x,y maximize 1 — ^ ^^^^'^ , there is still work to 
be done in analyzing the value of s*(r), and in particular its asymptotic 
behavior. For several examples in this paper, our method of writing ^J^^'-j^^ 
as a sum of non-negative terms will be well-suited for this. For all of the ex- 
amples in this paper, we do express s*(r) in terms of the distinct eigenvalues 
1, Ai, • • • ,Xd K: 

(1) 



(2) 



s*{r) =P(T > r), 

where T = X^iLi -^i ^^'^ the Xi are independent geometric random 
variables with probability of success 1 — Aj. 

These relations are useful for studying convergence rates and appeared ear- 
lier for certain one-dimensional problems (stochastically monotone birth- 
death chains started at 0) |DF| . [DSa] . and in [F3] for a higher dimensional 
problem (random walk on irreducible representations of Sn)- The current 
paper provides further examples, and revisits the results of [F3j using com- 
mutation relations. 

Section [3] reviews the concept of "down-up" Markov chains on branching 
graphs and describes some main examples to be analyzed in this paper. 
Aside from their intrinsic combinatorial interest, down- up chains are very 
useful. They were crucially applied in |Flj . |F4] to study asymptotics of 
characters of the symmetric group, and were recently used in |BQ2 ].[Pe] 
to construct interesting infinite dimensional diffusions. (Actually p302j . 
[Pe] use "up-down" chains instead of "down-up" chains; our methods apply 
to these too and it will be shown that they have the same convergence 
rate asymptotics). Convergence rate information about these chains is also 
potentially useful for proving concentration inequalities for statistics of their 
stationary distributions [C] . 

Section H] adapts Stanley's work on differential posets to the commutation 
relations (II. ip . These results are applied in Section [5] to study the down- 
up walk on the Young lattice. Here the stationary distributions are the so 
called z-measures, studied in papers of Kerov, Olshanski, Vershik, Borodin, 
and Okounkov (see [ KOVlj . |K()V2j . [502] . [B03], [Ol] and the references 
therein). In a certain limit these measures become the Plancherel measure 
of the symmetric group, and we obtain new proofs of results in [F3j . 
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Sections [6] analyzes down-up walk on the Schur lattice. We explicitly 
diagonalize this random walk, and use this to study total variation distance 
convergence rates. Similar ideas can be used to analyze down-up walk on 
the Jack lattice (see the discussion at the end of Section [6]) . The arguments 
in Section [6] do not require the use of commutation relations, though we do 
note some connections. 

Section [7| applies commutation relations to study down-up walk on the 
Kingman lattice. Here the stationary distribution depends on two parame- 
ters 9, a and when a = is the Ewens distribution of population genetics. 
The down-up walk is more "local" than the traditionally studied random 
walks with this stationary distribution, such as the random transposition 
walk when a = 0, ^ = 1; this could be useful for Stein's method. We show 
that the eigenvalues and separation distance do not depend on the parameter 
a, and prove order upper and lower bounds for the separation distance 
mixing time. Further specializing to the case 9 = 1 (corresponding to cycles 
of random permutations) we prove that for c > fixed. 



Note that in contrast to the random transposition walk, there is no cutoff. 

Section [8] treats other examples to which the methodology applies. This 
includes Bernoulli-Laplace models, subspace walks, and a Gibbs sampler 
walk on the hypercube. For most of these examples, the spectrum is known 
by other methods, and separation distance results (at least in continuous 
time) were described in |DSaj . However the hypercube example may be 
new, and in any case provides a nice illustration of how of our method for 
writing T^\y) ^ sum of non-negative terms allows one to determine the 
precise separation distance asymptotics. 



We will be concerned with the theory of finite Markov chains. Thus X 
will be a finite set and K a matrix indexed hy X x X whose rows sum to 
1. Let TT be a distribution such that K is reversible with respect to vr; this 
means that 7r{x)K{x, y) = Tr{y)K{y, x) for all x, y and implies that vr is a 
stationary distribution for the Markov chain corresponding to K. 

Define {f,g) = Ylxex f(^)9(^)'^i^) ^^^^ valued functions f,g on X, 
and let -/^^(vr) denote the space of such functions. Then when K is considered 
as an operator on L^(7r) by 



CXD 



n 




2. Probabilistic background 



Kf{x) :=Y,K{x,y)fiy) 



y 



it is self adjoint. Hence K has an orthonormal basis of eigenvectors fi{x) 
with Kfi{x) = Xifi{x), where both fi{x) and Aj are real. It is easily 
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shown that the eigenvalues satisfy —1 < A[x|-i < • • • < Ai < Aq = 1. 
If |Ai|, |A|x|_i| < 1, the Markov chain is called ergodic. 

2.1. Total variation distance. A common way to quantify the conver- 
gence rate of a Markov chain is using total variation distance. Given prob- 
abilities P, Q on X, one defines the total variation distance between them 
as 

It is not hard to see that 

\\P-Q\\=maK\PiA)-Q{A)\. 

Let be the probability measure given by taking r steps from the starting 
state X. Researchers in Markov chains are interested in the behavior of 

Lemma 12.11 is classical (see |DHj for a proof) and relates total variation 
distance to spectral properties of K. Note that the sum does not include 
i = 0. 

Lemma 2.1. 

1=1 

Lemma 12.11 is remarkably effective and often leads to sharp convergence 
rate results; we will apply it in Section [6l The main drawback with the 
bound in Lemma 12.11 is that one rarely knows all of the eigenvalues and 
eigenvectors of a Markov chain. In such situations one typically bounds the 
total variation distance in terms of max(|Ai|, |A|x|-i|) and the results are 
much weaker. 

2.2. Separation distance. Another frequently used method to quantify 
convergence rates of Markov chains is to use separation distance, introduced 
by Aldous and Diaconis [AD1| . [AD2] . They define the separation distance 
of a Markov chain K started at x as 

K-{x,yy 



s{r) = max 
y 



1 



7r(y) 

and the maximal separation distance of the Markov chain K as 

K-{x,yy 



s*(r) = max 
x,y 



1 



They show that the maximal separation distance has the nice properties: 

max I \Kl. — 7r| I < s*(r) 

X 

• (monotonicity) s*(ri) < s*(r2), ri > r2 

• (submultiplicativity) s*(ri + r2) < s*{ri)s*{r2) 
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For every e > 0, let n* be the smallest number such that s*{n^) < e. Many 
authors consider n* to be a definition of the separation distance mixing 

2 

time (see jPa] and references therein), and we also adopt this convention. 
Heuristically, the separation distance is ^ after n* steps and then decreases 

2 

exponentially. 

Lemma [2. 2 1 will give useful upper and lower bounds for n* . It is essentially 

2 

a reformulation of Corollary 2.2.9 of [Pa]. By the general theory in |AD2| . 
the random variable T in Lemma 12.21 always exists, but could be hard to 
construct. 

Lemma 2.2. Suppose that T is a random variable which takes values in the 
natural numbers and satisfies s*{r) = ¥{T > r) for all r > 0. Then 

KIT] 

< n*i < 2E[r]. 

2 2 

Proof. The upper bound follows since P(T > 2E[T]) < ^. For the lower 
bound, note that 



E[r] = ^P(r > r) = ^s*(r) < k + ks*{k) + ks*{kf + 

r>0 r>0 



1 - s*{k)' 



The inequality used monotonicity and submultiplicativity. Thus if A; < —5-^, 
then s*{k) > i, which completes the proof. □ 

For the next proposition it is useful to define the distance dist{x, y) be- 
tween x,y X as the smallest r such that K^(x,y) > 0. For the special 
case of birth-death chains on the set {0, 1, • • • ,d}, Proposition 12.31 appeared 
in [DF] and [EE]. 

Proposition 2.3. f]F3]J Let K be a reversible ergodic Markov on a finite 
set X . Let 1, Ai, • • • , A^ be the distinct eigenvalues of K . Suppose that x, y 
are elements of X with dist{x, y) = d. Then for all r > 0, 



K''{x,y) _ 



The relevance of Proposition 12.31 to separation distance is that one might 
hope that s*(r) is attained by x,y satisfying dist{x,y) = d. Then Proposi- 
tion [231 would give an expression for s*(r) using only the eigenvalues of K. 
Diaconis and Fill [DF| show (for s(r) when the walk starts at 0) that this 
hope is realized if IT is a stochastically monotone birth death-chain. In the 
current paper we give higher dimensional examples. 

Proposition 12.41 gives a probabilistic interpretation for the right hand side 
of the equation in Proposition 12.31 We use the convention that if X is 
geometric with parameter (probability of success) p, then ¥{X = n) = 
p(l — p)"~^ for all n > 1. 
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Proposition 2.4. Suppose that T = where the random variables 

Xi are independent, and Xi is geometric with parameter 1 — Aj G (0, 1]. // 
the A 's are distinct, then 



P(T>r) = ^A[ 



4 = 1 



y - A,- 



for all natural numbers r. 

Proof. By independence, the Laplace transform of T is 



E[e- 



-sTi 



i=l 



n 

i=l 



1-A, 
- A,; 



Since the Laplace transform of T is 

^[P(r > A: - 1) - P(r > A;)]e-"^ 



fc>i 



it suffices to substitute in the claimed expression for P(r > k) and verify 



that one obtains ni=i 



Observe that 



A;>1 i=l j^i 



At — Ao 



i=l fe>l 



Aj — Aj 



E ^ - TT 1 ~ Aj 
e« - Ai y Ai - A,- 



i=l 
d 



n^En 



k=l 



Afc . , . Aj — Aj 



Letting t = e^, note that the polynomial 

1=1 j^i J 

is of degree at most d — 1 in t but is equal to 1 when t = Aj for 1 < i < d. 
Thus the polynomial is equal to 1, and the result follows. □ 

Remarks: 

(1) Proposition 12.41 has a continuous analog where the geometries are 
exponentials |BSj . and the above proof is a discrete version of theirs. 

(2) For stochastically monotone birth-death chains with non-negative 
eigenvalues, Proposition 12.31 and Proposition 12.41 lead to the equality 
s(r) = P(T > r). Here s{r) is the separation distance of the walk 



COMMUTATION RELATIONS AND MARKOV CHAINS 



7 



started at 0, and T is the sum of independent geometries with pa- 
rameters 1 — Aj, where the Aj's are the distinct eigenvalues of the 
chain not equal to 1. This equality was first proved in |DF] using 
the theory of strong stationary times, and was beautifully applied to 
study the cutoff phenomenon in |DSaj . 

2.3. Cut-off phenomenon. Since the term is mentioned a few times in 
this article, we give a precise definition of the cutoff phenomenon. A nice 
survey of the subject is [Dj; we use the definition from [Salj . Consider a 
family of finite sets X„, each equipped with a stationary distribution 7r„, 
and with another probability measure pn that induces a random walk on 
Xn- One says that there is a total variation cutoff for the family 7r„) if 
there exists a sequence of positive reals such that 

(1) lim^^oo^n = oo; 



(2) For any e G (0, 1) and r„ = [(1 + e)t„J, lim„_oo - 7r„.|| = 0; 

(3) For any e G (0, 1) and r„ = [(1 - e)t„J, lim„_oo |K" " ^nll = 1- 
For the definition of a separation cutoff, one replaces HpJ^" — TTnll by s*(r„). 



This section recalls the construction of down-up Markov chains on branch- 
ing diagrams and describes some main examples to be studied later in the 
paper. Down-up chains appeared in [FT] and more recently in [B02| : they 
are obtained by composing down and up Markov chains of Kerov [K] . 

The basic set-up is as follows. One starts with a branching diagram; that 
is an oriented graded graph F = U„>oF„ such that 

(1) Fq is a single vertex 0. 

(2) If the starting vertex of an edge is in Fj, then its end vertex is in 



(3) Every vertex has at least one outgoing edge. 

(4) All Ti are finite. 

For two vertices A, A G F, one writes A ^ A if there is an edge from A 
to A. Part of the underlying data is a multiplicity function k(A, A). Letting 
the weight of a path in F be the product of the multiplicities of its edges, 
one defines the dimension d\ of a vertex A to be the sum of the weights over 
all maximal length paths from to A; dim{^) is taken to be 1. 

A set {Mn} of probability distributions on F„ is called coherent if 



Letting {Mn} be a coherent set of probability distributions, one can define 
the "up" Markov chain which transitions from r G F„_i to p G F„ with 



if r is distributed from M„_i, then p is distributed from M„. Similarly, 
one can define the "down" Markov chain which transitions from A G F„ to 



3. Down-up Markov chains 




probability 



drM„(p)K{T,p) 

dpMn-iir) 



This preserves the set {Mn} in the sense that 
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r G r„_i with probability 




This also preserves {M„}. Composing 



these Markov chains by moving down and then up, one obtains the "down- 
up" Markov chain in the level r„ of the branching diagram. This moves 
from A to p with probability 



This Markov chain has M„ as its stationary distribution and is in fact re- 
versible with respect to M„. 

The reader may wonder whether there are interesting examples of coher- 
ent probability distribution on branching diagrams. In fact there are many 
such; see the surveys [K] and |BQ1| . To make the above definitions more 
concrete, we now describe two examples which are analyzed in this paper 
(Young and Kingman lattices). We will also analyze down- up walk on the 
Schur and Pascal lattices, but define them later. 

Example 1: Young lattice 

Here r„ consists of all partitions of size n, and (identifying a partition 
with its diagram in the usual way [Mac]) a partition A of size n is adjoined to 
a partition A of size n+1 if A can be obtained from A by adding a box to some 
corner of A. The multiplicity function «;(A, A) is equal to 1 on each edge. 
The dimension function d\ has an algebraic interpretation as the dimension 
of the irreducible representation of the symmetric group parameterized by 
A, and there is an explicit formula for d\ in terms of hook-lengths |Sag| . 

An important example of a coherent set of probability distributions on 
the Young lattice is given by the so called z-measures. This is defined using 
two complex parameters z, z' such that zz' {0, —1, —2, • • • }, and assigns a 
partition A weight 



Here c(6) = j — i is known as the "content" of the box b = with row 
number i and column number j. In order that M„ be strictly positive for all 
n, it is necessary and sufficient that (z, z') belongs to one of the following 
two sets: 

• Principal series: Both z, z' are not real and are conjugate to each 



• Complementary series: Both z, z' are real and are contained in the 
same open interval of the form (m, m + 1) where m G Z. 

The z-measures are fundamental objects in representation theory (see 
[KOVlj , |K0V2j ) and become the Plancherel measure of the symmetric group 
in the limit z,z' oo. 

Example 2: Kingman lattice 

Here the branching diagram is the same as the Young lattice, but the 
multiplicity function k(A, A) is the number of rows of length j in A, where A 




Y{^^^{z + c{h)){z' + c{b)) dj 
zz'{zz' -|- 1) • • • {zz' -|- n — 1) n! 



other. 
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is obtained from A by removing a box from a row of length j. The dimension 
function has the exphcit form d\ = where / is the number of rows of 

A and Aj is the length of row i of A. 

The Pitman distributions form a coherent set of probability distributions 
on r^. These are defined in terms of two parameters 6 > and < q < 1. 
The Pitman distribution assigns A probability 

Here mi{\) is the number of parts of A of size i. When a = 0, this becomes 
the Ewens distribution of population genetics. Further specializing to a = 
0, = 1, gives that M„(A) is equal to the chance that a random permutation 
on n symbols has cycle type A. 

4. Commutation relations 

It is assumed that the reader is familiar with the concept of partially 
ordered sets, or posets for short. Background on posets can be found in 
Chapter 3 of the text [Stlj . All posets considered here are assumed to 
be locally finite (every interval [x^ y] of P consists of a finite number of 
elements) and graded (every maximal chain from a point x to a point y has 
length depending only on a;,y). It is also assumed that P has an element C) 
satisfying x > for all x £ P. 

Given a locally finite poset P and x £ P, let CP denote the complex 
vector space with basis P, and let CP„ denote the subspace of CP spanned 
by the rank n elements (the rank of an element x is the length I of the longest 
chain xq < xi < • • • < x; = x in P with top element x). Write x y \i y 
covers x in the poset P. Stanley |St2j defined up and down operators C/, D 
by the condition that for x G P, 

Ux= ^ y , Dx= ^ y. 

y.x/y y-y/x 

These operators can be extended by linearity to CP. For A : CP i-^ CP, let 
An denote the restriction of A to CP„. Notation such as ABn is unambiguous 
since A{Bn) and [AB)n have the same meaning. Linear transformations 
will operate right-to- left, e.g. DUv = D{Uv), and / will denote the identity 
operator. 

Stanley (loc. cit.) defined a locally finite, graded poset with element to 
be differential if its up and down operators satisfy the commutation relation 

DU-UD = rl 

for some positive integer r. He determined the spectrum and eigenspaces 
(though typically not eigenvectors) of the operator UDn- In the follow-up 
paper |St3] . Stanley extended his ideas to the commutation relation 

Dn+lUn — Un-lDn = Tnln 
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where the r„'s are integers. 

We study the more general case that Un ■ CP„ ^ CP„+i and Dn : CPn ^ 
CPn-i are hnear operators satisfying the commutation relation (1.1) of the 
introduction: 

Dn+lUn = anUn-lDn + bnJn, 

where a„, 6„ are real numbers. The results we need do not all appear in |Foj 
(who also studied this relation), so we briefly give statements and proofs. 
This serves both to make the paper self-contained and to illustrate the power 
of Stanley's methods. 

Theorem 14.11 determines the spectrum of C/D„. It can be easily derived 
from Theorem 1.6.5 of [Fo]. 

Theorem 4.1. Suppose that the commutation relations (1.1) hold and that 
cin > for all n> 1. Let pj denote the number of elements of P of rank j. 
Then the eigenvalues of UDn are 

multiplicity pn — Pn-i 

YJj=i bj nfc=i+i "fc multiplicity pi - pi^i {0 < i < n - I) 
In particular, if bi = 1 — ai for all i, these become 
multiplicity pn — Pn-i 

rn-l 



1 — nfc=i multiplicity pi — pi^i (0 < i < n — 1) 

Proof. The proof is by induction on n. Let Ch{A) = Ch{A,X) be the 
characteristic polynomial det{XI — A) of an operator A. Since Ch{U-iDQ) = 
A, the theorem is true for n = 0. Suppose that A:V^W and B : W V 
are linear transformations on finite dimensional vector spaces V and W and 
that dim{V) = v and dim{W) = w. Then (by [Wkj, Ch.l, Sec. 51), 

Ch{BA) = X'"-'"Ch{AB). 

Applying this to and [/„ gives that 

Ch{UnDn+i,X) = XP-+^-P-Ch{Dn+iUn,X) 

= XP-+^-P-Ch{anUn-lDn + bnInA) 

= XP"+^-P-Ch{anUn-iDn, X - 6„) 
l\ an J 

Hence is an eigenvalue with multiplicity at least Pn+i — Pm and if is 
an eigenvalue of Un-iDn of multiplicity m^, then a„Afc + 6„ is an eigenvalue 
of UnDn+i of multiplicity at least nik. This implies the eigenvalue formula 
in terms of the a, b variables. If one sets bi = 1 — ai for all i, then the sum 
telescopes, yielding the second formula. □ 

To compute the eigenspaces of U Dn , the following lemma is useful. These 
eigenspaces won't be needed elsewhere in the paper, although knowing them 
could prove useful in the search for eigenvectors, which by Lemma 12.11 are 
useful for the study of total variation distance convergence rates. 
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Lemma 4.2. Suppose that the commutation relations (1.1) hold, withbo = 1 
and an,bn > for all n > 1. Then the maps Un are injective and the maps 
Dn+i are surjective. 

Proof. The case n = is clear since DiUq = Iq. For n > 1, recall the 
commutation relation 

By Theorem 14.11 and the assumption that a„ , 6„ > for all n > 1, it follows 
that all eigenvalues of Un-iDn are non-negative. Thus all eigenvalues of 
Dn+iUn are positive. Thus is not an eigenvalue and the result follows. □ 

Theorem 4.3. Suppose that the commutation relations (1.1) hold, with 
6o = 1 and an,bn > for all n > 1. Let En[\) denote the eigenspace of 
UDn corresponding to the eigenvalue A. 

(1) En{0) = ker{Dn). 

(2) i?n(E = U-^EM- 

Proof. The first assertion is clear from Lemma 14.21 To prove the second 
assertion, we show that 

(n-2 n-2 \ /n-1 n-1 

j=i k=j+l I \j=i fc=j+l 

By Theorem 14.11 the multiplicity of Yl^Zt bj 11^=^+1 '^fc as an eigenvalue of 
UDn-i is the multiplicity of X]j=/ nfe=j+i '^fe as an eigenvalue of UDn. 
Thus since Un-i is injective (Lemma 14. 2p . it is enough to check that 

(n-2 n-2 \ /n-1 n-1 

j=i k=j+l I \j=i /c=j+l 

So suppose that v E En~i{J2^~f bj YYkZ'j+i a-k). Then commutation relation 
(1.1) yields that 

UDn{Un-lv) = an-lUn-l{Un-2Dn~lv) + bn-lUn-lV 

n-2 n-2 

= On-l^^^i n (^k ■ Un-lV + bn-lUn-lV 
j=i k=j+l 
n—1 71—1 

j=i k=j+l 

□ 

Another tool we need is an expression for (UD)!^ as a linear combination 
of {U^D^)n, extending that of [St2] for the case of differential posets. 
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Lemma 4.4. Suppose that the commutation relations (1.1) hold. Then 

n n n 

j=n—k+l j=n—k+l l=i+l 

for all 1 < k < n. In particular, if bi = 1 — ai for all i, this becomes 

n / " \ 



j=n— fc+1 



j=n— fc+1 



Proof. This is straightforward to verify by induction on fc, writing = 
D{D^-Wn) and then using commutation relation (1.1). □ 

Now the desired expansion of {UDY^ can be obtained. We remark that 
for the examples studied in this paper, the coefficients in the expansion will 
be non-negative. 

Proposition 4.5. Suppose that the commutation relation (1.1) holds. Then 

n 

{UD)l = Y,Mr,k){U''D% 

k=0 

where An{r,k) is determined by the recurrence 



n-l 



71—1 n— 1 



An{r,k) = An{r-l,k-l) J] + A„(r - 1, /c) ^ bj J| ai 

j=n—k+l j=n—k l=j+l 

with initial conditions ^^(0,0) = 1 and A„(0,m) = for m ^ 0. In partic- 
ular, if < tti < 1, bi = 1 — tti for all i, then the recurrence becomes 



n — 1 



n-l 



An{r,k)=An{r-l,k-l) J] aj + An{r - 1, k) I 1 - 



j=n—k+l 

and all An{r, k) are non-negative. 



j=n—k 



Proof. The proposition is proved by induction on r. The base case r = is 
clear. First applying the induction hypothesis and then Lemma 14.41 yields 
that (UD)^ is equal to 



Y,Mr-l,k)U''D''UDn 

k=0 
n 



k=0 



n-l 



n-l n-l 



j=n—k j=n—k l=j+l 
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This implies the recurrence 

n— 1 n— 1 n—1 

Anir,k) = Anir -l,k-l) Oj + A„(r - 1, /c) ^ bj ai, 

j=n—k+l j=n—k l=j+l 

and the rest of the proposition follows immediately. □ 

As a final result, we give a generating function for the j4„(r, k) of Propo- 
sition 14.51 By comparing with Theorem 14.11 one sees that the eigenvalues of 
UDn appear in the generating function. 

Proposition 4.6. For k >0 set Fk{x) = ^^^^x'^ An[r,k), where An{r,k) 
was defined in the statement of Proposition |^.4 Then 

Fk{x) 



^^ntinpUi". 



nLi (i - ^ TJ'jJ.-i bj nr=/+i a/ 



In particular, if bi = 1 — a,, for all i, then 



Fk{x) 



n-=i i-^(i-n-=n- 



lj=n-i '^3 



Proof. Clearly Fq{x) = 1. For A; > 1, multiply both sides of the recurrence 
of Proposition 14.51 by x^ and sum over r > to obtain that 

Fk{x) = An{Q,k) + Y,x' Mr,k) 

r>l 

n-l 

= J];xM„(r- 1,/c- 1) W aj 

r>l j=n—k+l 

n—l n—1 

+ Y,x''An{r-l,k) bj Yl ai 

r>l j=n—k /=i+l 

n—1 n—1 n—1 

= xFk_i{x) Yi aj+xFk{x) ^ bj J| ai. 

j=n—k+l j=n—k l=j+l 



Thus 



Fk{x) 



xFfc_i(x)n"=n-fc+l°j 
1 V— vn— 1 I Tfi— 1 ' 

^-X2^j=n-kbj[[i=j+iai 



-vn— 1 I TT'*~1 ' 
jj=n-k 1 h=j+l 

and the result follows by induction. □ 



5. The Young lattice 

The purpose of this section is to use commutation relations to study 
separation distance for down-up walk on the Young lattice. At the end of 
the section, it is shown that the same asymptotics hold for up-down walk. 



14 



JASON FULMAN 



The setting is that of Example 1 in Section [3l Thus the down-up walk is 
on partitions of size n, and the chance of moving from A to p is equal to 



and the z-measure is its stationary distribution. Here p/r denotes the box 
of p not contained in r, and c{h) = j — i is the "content" of the box b = 
We remind the reader that it is assumed that either z' = z with z G C — M, 
or that z, z' are real and there exists m G Z such that m < z, z' < m -\- 1. 

In the limiting case that z,z' oo, the stationary distribution becomes 
Plancherel measure of the symmetric group. The paper |F1] determined 
the eigenvalues and an orthonormal basis of eigenvectors for down-up walk 
in this case. Then sharp total variation distance convergence rates for this 
random walk were obtained in [F2], and separation distance asymptotics 
were derived in |F3] . We give new proofs of some of these results using 
commutation relations, and generalizations to the setting of z-measures. 

To begin, we define operators ■ CP„ ^ CPn-i and C/„ : CP„ i— > CP„+i 
as the linear extensions of 



Note that by the hypotheses on z, z' , the coefficient of any partition in Dn{\) 
or Un{X) is non-negative. 

The following lemma is equivalent to Lemma 4.2 of |B02j and is essentially 
due to Kerov (see lOl]). 

Lemma 5.1. 



with a„ = 1 n — o.'nd 6„ = 1 H P;— . 

Let A be the diagonal operator on CP which sends X to dx- A. Then it is 
clear that the down-up walk on Young's lattice corresponds exactly to the 
operator ^{AUDA~^)n. 

In Corollary [521 pU) denotes the number of partitions of j. By conven- 
tion, p{0) = 1. 

Corollary 5.2. The eigenvalues of the down-up walk on the nth level of 
the Young lattice are ^ (^ ^ ^ zz'+n- (0 < i < n), with multiplicity equal to 
p{n — i) — p{n — i — 1). 

Proof. This is immediate from Theorem 14. H Lemma |5.H and the fact that 
the down-up walk on Young's lattice is given by ^{AUDA~^)n- □ 

Remark: It is not difficult to see that p{n — i) — p{n — i — 1) is equal to 
the number of partitions of n with i I's. Indeed, using the notation that 





Dn+lUj 



a-nUn-lDn + 



n 
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[n"]/(n) is the coefficient of in f{u), one has that 

p{n-i) -p{n-i-l) = K"*] JJ(l-n^)-i - [n"-^-i] J|(l-u^)-i 

i>i i>i 

which is the number of partitions of n — i with no I's. 



Proposition 15.31 is crucial for determining where the maximal separation 
distance of down-up walk on Young's lattice is attained. Its statement uses 
the notation that if B : CP ^ CP, then B[^, X] is the coefficient of A in 

Proposition 5.3. Let tt{X) be the z-measure evaluated at X, and let r be a 
non-negative integer. Then the quantity 

{^AUDA-^nf,,X] 
niX) 

is minimized (among pairs of partitions of size n) by ^ = (n),A = (1") or 
/i = (l"),A = (n). 

Proof. Lemma l5. II and Proposition 14.51 give that 

-(A) "^/^^'''^ vr(A) ' 
where An{r, k) is determined by the recursion of Proposition 14. 5[ Thus 
{}-AUDA~^Y[fi,X] _ 1 ^ dxAn{r,k)U''D''[is,X] 

The proposition follows immediately from three claims: 

• All terms in the sum are non-negative. Indeed, since bn > for 
n > and a„ > for n > 1, the recursion for j4„(r, k) implies that 
An{r,k) > 0. Noting that U,D map non-negative linear combina- 
tions of partitions to non-negative linear combinations of partitions, 
the claim follows. 

• If /i = (n),A = (1") or /i = (1"),A = (n), then the summands for 
< k < n — 2 vanish. Indeed, for such k it is impossible to move 
from the partition /i to the partition A by removing k boxes one at 
a time and then reattaching k boxes one at a time. 
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• The k = n — 1 and k = n summands are independent of both fj, and 
A. Indeed, for the k = n — 1 summand one has that 

dxAn{r,n-l)U^-^D^-^[f,,X] 

n^d^7r(A) 
dxAnir,n-l)U^-'[{l),X\ 

n''7r{\) 

dlAn{r,n-l)Yl tex (z + c(6))(z' + c(6)) 

~ n''"(zz' + 1) • • • (zz' + (n - l))7r(A) 

n\An{r, n — 1) 
rf 

The first equahty used the fact that there are ways to go from [i 
to (1) by removing a box at a time. The second equahty used the 
fact that all d\ ways of transitioning from (1) to A in n — 1 upward 
steps give the same contribution to ?7"~^[(1), A]. 

A similar argument shows that the k = n summand is equal to 

n\An(r,n) 
n'" 

□ 

Corollarv 15.41 gives an expression for maximal separation distance. 

Corollary 5.4. Let s*{r) he the maximal separation distance after r itera- 
tions of the down-up chain K on the nth level of the Young lattice. Then 
s*{r) = P(r > r), where T is a sum of independent geometries with param- 

etersl-^^[^q^) for^<^<n-2. 

Proof. By Proposition 15.31 s*{r) = 1 ^^i^^ — • Proposition 15.21 the 

down-up walk has n distinct eigenvalues, namely 1 and ^ {^ ^^zt'^+n-i^ '^ 
< i < n — 2. Since the distance between (n) and (l") is n — 1, the result 
follows from Propositions 12.3 1 and 12.41 □ 

Theorem 15.51 gives a precise expression for the asymptotics of separation 
distance in the special case that z,z' — > oo. Then the stationary distribu- 
tion is Plancherel measure of the symmetric group, and these asymptotics 
were obtained earlier in |F3| . Here we present a new proof which involves 
determining the numbers An{r,k). This technique is likely to prove use- 
ful for other problems; in particular, we apply it again later in this paper 
(Proposition ES]). 

Theorem 5.5. Let s*{r) be the maximal separation distance after r itera- 
tions of the down-up walk on the nth level of the Young lattice, in the special 
case that z, z' — > oo. 

(1) s*(r) = 1 — "■■^(^'^" ^) _ "!-5^n") ^ yjiiQfg^ S{r,k) is a Stirling number 

of the second kind (i.e. the number of partitions of an r-set into k 

blocks). 
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(2) For c fixed in M and n ^ co, 

s*{n log(n) + cn) = 1 - e-'~\l + 6"'=) + O (^^^ 

Proof. For the first assertion, the proof of Proposition 15.31 gives that 

n\An{r,n-l) n\An{r,n) 
s r = 1 . 

^ ' n'" ri' 

The recurrence in Proposition 14.51 is 

A„(r, k) = An{r -l,k-l) + kAn{r - 1, k) 

with initial conditions An{0,m) = 5o,m.- The solution to this recurrence is 
An{r,k) = S{r,k) (see also Proposition 4.9 of [St2j ). and the first assertion 
follows. 

Let P{n, r, k) denote the probability that when r balls are dropped uni- 
formly at random into n boxes, there are k occupied boxes. It is straight- 
forward to see that P{n,r,k) = ^ili^Mi). Indeed, occupying k boxes using 
r balls is equivalent to forming an ordered set partition of { 1 , • • • , r} into k 
blocks and then choosing k of the n boxes. Thus, 

s*{r) = 1 — P{n, r, n — 1) — P{n, r, n). 

Now we use asymptotics of the coupon collector's problem: it follows from 
Section 6 of [CDMj that when n log(n) + cn balls are dropped into n boxes, 
the number of unoccupied boxes converges to a Poisson distribution with 
mean e~^, and that the error term in total variation distance is 0( ^°^^"^ ). 
The chance that a Poisson random variable with mean takes value not 
equal to or 1 is 1 — e"*^ (1 + e~'^), which completes the proof. □ 

For general values z, z' it is not evident how to obtain results as clean as 
Theorem l5.5l However Proposition 15. 61 gives explicit upper and lower bounds 
for the separation distance mixing time. For z, z' fixed and n growing, these 
are both order n?. 

Proposition 5.6. Let n* be the separation distance mixing time of down- 

2 

up walk (corresponding to z-measure) on the nth level of the Young lattice 

2 

2 

zz' = 1 then 



Then ^ < n\ < 2E[T], where T is as in Corollary \5.4\ Moreover, if 



i=2 ^ ^ 



and if zz' ^ 1 then 



nizz' + n-l)^ (2{n + zz'-l)\ 

1 + '- bg -^ < E T 

I- zz' ^ V n{zz' + 1) ) - ^ ^ 

n(zz' + n — 1) , f n + zz' — 1 
< ^ log ' 



1 — zz' \ n{zz') 
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Proof. By Lemma 12.21 < < 2E[T]. Linearity of expectation gives 

2 

that 

n-2 



E\T] = y 



^ n \ 2z'+n— 1 

When zz' = 1, 

n—2 ^ n—2 ^ " 1 / 2 \ 

E[T]=y =n^y = n2 y i ~ n2 ( ^ - 1 ) . 

For zz' 7^ 1, the fact that '^^(^^/^^^^/^ monotone increasing for i E 
[0, n — 1] gives that 

rn—2 Y rn—1 ^ 

1 + / 7 rdt < E[T] < / -. :-dt. 

Jo 1 _ 1 ( 2^'+2n-f-l \ Jq I _ L ( zz'+2n-t-l ^ 



n \ zz'+n—l I n \ zz'+n—l 

Consider the upper bound on E[r]. Since zz' 7^ 1, it is equal to 
nlzz' + n-l) f I 1 , 

dt 



1 — zz' Jq \t — n t — (n + zz' — 1) 
n(zz' + n — 1) , f n + zz' — 1 



- 1) , fn + zz'-l\ 

log 

z' \ n(zz') J 



1 - zz' 

Similarly, since zz' ^ 1, the lower bound on E[r] is equal to 

„(..- + „-!) I 

1-zz' Jq \t-n t-{n + zz'-l)/ 

nizz' + n — 1) / 2in + zz' — 1] 
= IH ; log 



1 — zz' \ n{zz' + 1) 



□ 



To close this section we prove Proposition l5.7i It implies that the up-down 
and down-up walks have the same convergence rate asymptotics. 

Proposition 5.7. Let s^j~i^{r) he the maximal separation distance after r 
iterations of the down-up chain (corresponding to z-measure) on the Young 
lattice, and let s*j-)jj^{r) be the corresponding quantity for the up-down chain. 
Then 

for all n > 1, r > 0. 

Proof. Using the notation of Proposition [5T3l the up-down chain corresponds 
to the operator ^^^ADUnA^^ . Lemma |5. II implies that 

{^,ADUnA-^nf,,X] _ l_^fr\j ^,^i^i{^AUD^A-')%,X] 

lY ^ 



vr(A) in^\r^^\l) vr(A) 
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where = 1 — ^zKn ^^'^ &n = 1 + zz^+n ■ Hence Proposition 15.31 gives that 
this quantity is minimized by /x = (n), A = (1") or fi = (1"), A = (n). 

By Corollary 15.21 and Lemma 15. H the distinct eigenvalues of the up-down 
chain are 1 and tj := where 1 < j < n — 1. Hence Proposition 



T3] gives that 

tk 



f 1 - 



l<fc<n-l 

On the other hand, applying Proposition 12.31 to the down-up chain gives 
that 



n— 1 



7=0 fc#J 

0<fc<n-l 



Since to = 0, this becomes 



Efe)'" n (^) n 

J = l fc^. ^'^^ i = l fcT^. ^^-^ 

0<fc<n-l l<fe<n-l 

as desired. □ 

6. The Schur lattice 

In this example the underlying lattice is the Schur lattice. This is the 
sublattice of Young's lattice consisting of the partitions of n into distinct 
parts. We show that commutation relations can be used to compute the 
spectrum of down-up walk on the Schur lattice, but our approach does not 
determine the separation distance convergence rate (the obstacles are de- 
scribed in the second remark after Proposition 16. 2|) . We do however give a 
complete diagonalization of the Markov chain, and use it to study the total 
variation distance convergence rate. The upper bound derived here is in fact 
quite sharp and there is a cutoff at ^nlog(n). We omit the rather involved 
proof of a matching lower bound but give a careful statement and explain 
the proof technique in the remarks after Theorem 16. 4[ 

It will be convenient to let DP{n) denote the set of partitions of n into 
distinct parts and OP{n) denote the set of partitions of n into odd parts. 
Using the terminology of Section [3l there is a coherent set of probability 
distributions on the Schur lattice called the shifted Plancherel measures. 
The nth measure chooses a partition A G DP{n) with probability 

where is the number of parts of A and g\ is the number of stan- 
dard shifted tableaux of shape A ( [HH] , [Mac] ) . This measure is of inter- 
est to researchers in asymptotic combinatorics and representation theory 
[B],[Mat],[TWl, d]. 
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In the terminology of Section [3l it is known (see for instance [BOlj ) that 
the dimension of A G DP[n) is equal to g\. Hence the down-up chain on 
the set DP{n) transitions from \ to p with probability 

An application of this Markov chain appears in [F4| . However nothing seems 
to be known about its convergence rate. 

We will diagonalize this chain (determining eigenvalues and eigenvectors) . 
Before doing this we note that commutation relations can also be used to 
derive its eigenvalues. The key is the following observation of Stanley |St3| . 
He defined down and up operators D, U for the Schur lattice by: 

D{X) = Y,f^, U{X) = 2 E E ^' 

i(/j)=i(A) i(i/)>i(A) 

and showed that they satisfy the commutation relation 

(6.1) Dn+lUn = Un-lDn + In 

for all n > 0. 

In Proposition l6.2l p*{j) denotes the number of partitions of j into distinct 
parts. 

Proposition 6.2. The eigenvalues of the down-up walk on the Schur lattice 
are ^ (0 < i < n), with multiplicity equal to p*{n — i) — p*[n — i — 1). 

Proof. Let A be the diagonal operator on CP which sends A to g\-\. It is eas- 
ily seen that the down-up chain is equivalent to the operator -{AU DA~^)n- 
The result now follows from commutation relation ()6.ip and Theorem 14.11 

□ 

Remarks: 

(1) It is well known that \DP{n)\ = \OP{n)\. Using generating functions 
as in the remark after Proposition 15.21 one can show that p*{n — i) — 
p*(n — i — 1) is equal to the number of odd partitions of n with i 
parts equal to 1. This also follows by comparing Proposition 16.21 
with Proposition 16.31 below. 

(2) From the previous remark, it is easily seen that the number of dis- 
tinct eigenvalues of UDn is n — 2 for large enough n (an odd partition 
of n can't have i parts of size 1 for i = n — l,n — 2,n — 4). However 
the diameter of down-up walk on the Schur lattice can be smaller 
than n — 3 (for n = 8 it is 4) . This blocks the use of Proposition 12.31 
and also complicates the analysis of where the maximal separation 
distance is attained, as the proof of Proposition 15.31 does not carry 
over. 

To upper bound the total variation distance convergence rate, the fol- 
lowing diagonalization of the down-up walk is crucial. The eigenvectors are 
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given in terms of symmetric functions, more precisely in terms of which 
is defined as the coefficient of the Hall-Littlewood polynomial Px{x;—1) in 
the power sum symmetric function Pfj.{x). The reader unfamiliar with these 
concepts can either consult Chapter 3 of [Macj (which calls these coefficients 
X^{—1)), or can just proceed to Theorem l6.41 We also use the notation that 

= i"^'-^^^ mi{fj,y. , where mi{^) is the number of parts of /i of size i. This 
is the number of permutations which commute with a fixed permutation of 
cycle type n. 

Proposition 6.3. (1) The eigenvalues of down-up walk on the Schur 
lattice are parameterized by fi £ OP{n) and are "^^^^^ , where mi(/x) 
is the number of parts of of size 1. 



The functions V'/^(A) = ^ ^ 2^-'-^ '^ '^^^ ^ corresponding basis of 
eigenvectors, orthonormal with respect to the inner product 

(/l,/2)= J2 /l(A)/2(A) ^, 

XeDP(n) 



Proof. It follows from Lemma 5.6 and Corollary 5.11 of |F4| that the V'/^ are 
an orthonormal basis of eigei 
operator J(„_i^i), defined by 



an orthonormal basis of eigenvectors with eigenvalue "^^^^2~^ ^ certain 



The proposition follows from the claim that the chance that the down-up 
chain moves from A to p is equal to 

(n-2)J(„_i,i)(A,p) 2^ 

1 dA,p 



n n 



where 5\^p is 1 if A = p and vanishes otherwise. For the case that A 7^ p, 
the claim follows from the statement of Proposition 5.9 of |F4j . and for the 
case A = />, it follows from the proof of Proposition 5.9 and Lemma 5.3 of 
[F4l. □ 



Finally, we use the diagonalization to study total variation distance for 
down-up walk on the Schur lattice. 

Theorem 6.4. Let denote the distribution of the down-up walk on the 
Schur lattice started from (n) after r steps, and let vr denote the shifted 
Plancherel measure. For r = ^nlog(n) + cn with c > 0, 
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Proof. The diagonalization of the down-up walk, together with Lemma l2. II 
and the facts [Mac] that = 1 and xj]^^ = 1 for all gives that 



mi(/i) 



2r 



m60P{7i) 

i=l ^ ^ MeOP(n) A* 



Letting [n"']/(n) denote the coefficient of in /(w), the cycle index of the 
symmetric group (reviewed in Chapter 4 of |Wi] ) yields that 



n! 



m]^(;j)=i 



n! 



i!2^ 



- n" 



n 

m>3 
odd 



il2 



m>l 
odd 



i!2"- 



n! 



i!2"- 



^ (1 - ti)e2'' 
^ (-2^ (-2)J 
i=o -^^ i=o -^^ 



It is easily checked that 



.2)J (-2)J 



i=o 



j=0 



vanishes if n — i = 1, 2, 4 and when n — i > is at most 2/3. 
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Thus 

- y 



liiT'^-TrlP < 



n-3 / . X 2r , 



< 



6 ^ \nj i\2^- 

1=1 

n 

J .2r 



(n-7)!2i 
i=3 ^ ■^'^ 



1 " 



< 



< 



6 ^ (n - ?)!2i 
i=3 

1 X - n] 

6 ^ (n - ?•)! 2r 

j=3 ^ -^^ 

1 >A n! e-^'^J 
6 ^ (n - j)! nJ2J 

1 " 

6 ^ 2J 

J =3 

g-6c 

48(1 - e-2c/2) 

c-6c 



_g2r-log(l-j/?i) 



< 



24 

Taking square roots completes the proof. □ 
Remarks: 

(1) One can prove that there are positive universal constants A, B such 
that for all c > and r = ^nlog(n) — cn with n large enough 
(depending on c), 



The proof method is analogous to that used in [F2j for the case of 
Plancherel measure of the symmetric group, but the combinatorics 
is more tedious. One can compute the mean and variance of the 
eigenfunction ■0(3 j^n-s) under both vr and the measure , and then 
deduce the lower bound from Chebyshev's inequality. 
(2) From commutation relation (|6.ip . the results in this section give 
(in the notation of Proposition 16. 3p that up-down walk on the Schur 
lattice has eigenvalues '^^^^^^^ and the same eigenfunctions as down- 
up walk. Arguing as in Theorem 16.41 gives that the walks have the 
same convergence rate asymptotics. 

To conclude this section, we mention that the techniques in it can be used 
to analyze total variation distance convergence rates for down-up walk on the 
Jack lattice. Here the stationary distribution is the so-called Jackc measure 
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on partitions, which in the special case a = 1 gives the Plancherel measure 
of the symmetric group. The importance of Jack^ measure is discussed in 
Okounkov |02| . and some results about it appear in [BQ4] and [F5j . In 
particular, Proposition 6.2 of [F5j explicitly diagonalizes down-up walk on 
the Jack lattice. The eigenvalues turn out to be independent of a and are 1 
and ^ for < i < n — 2. The eigenvectors are the coefficients of power sum 
symmetric functions in the Jack polynomials with parameter a. Further 
details may appear elsewhere. 



7. The Kingman lattice 

This section uses commutation relations to study down-up walk on the 
Kingman lattice. The stationary distribution is the Pitman distribution 
with parameters 6, a where > and < a < 1 (Example 2 in Section 
[3]). We show that the eigenvalues and separation distance do not depend 
on a and prove order upper and lower mixing time bounds. Very pre- 
cise convergence rate results are given when 6 = 1. This is probably the 
most interesting case, since when a = 0,9 = 1 the stationary distribution 
corresponds to the cycle structure of random permutations. 

The down-up walk studied in this section is more "local" the the random 
transposition walk, in the sense that the underlying partition is changed by 
removing a single box and then reattaching it somewhere. In the random 
transposition walk, the change is more violent: two cycles can merge into 
one cycle or a single cycle can be broken into two cycles. Local walks tend 
to be more useful for Stein's method than non-local walks (see [R] for some 
rigorous results in this direction), and this down-up walk was described 
in Section 2 of [Flj in the context of Stein's method. The recent paper 
|Pe] applies down-up walk on Kingman's lattice to define a new family of 
infinite dimensional diffusions, which includes the infinitely- many-neutral- 
alleles-diffusion model of Ethier and Kurtz. 

Now we begin the analysis of the down-up chain corresponding to the 
Pitman distribution with parameters 6 > and < a < 1. By the formulas 
in Section [3l one sees that the down chain removes a box from a row of 
length j with probability ^"^^^^^ and that the up chain adds a box to a row 
of A of length k > 1 with probability or to a row of length with 

probability where /(A) is the number of parts of A. In the biological 

context (a = 0), the rows of A could represent the count of individuals of 
each type in a population. Then the down move corresponds to the death 
of a random individual, and the up move corresponds to a birth (which is 
the same type as the random parent or a new type with probability g^)- 

Let P be the poset of partitions with the same partial order as in King- 
man's lattice, where we disregard edge multiplicities; this is the same partial 
order as in Young's lattice. It is natural to define operators D, U : CP CP 
as follows. The coefficient of r in -D„(A) is defined to be the probability that 
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from A, the down-chain transitions to r. The coefficient of A in ?7„(A) is de- 
fined to be the probabiUty that from A, the up-chain transitions to A. Thus 
the down-up walk on Kingman's lattice arising from Pitman's distribution 
is just the operator C/Z)„. 

The following commutation relation is crucial. Note that a closely related 
commutation relation appears in |Pe| . 

Proposition 7.1. Consider down-up walk on the Kingman lattice with pa- 
rameters 9 > and < a < 1. Letting an = ("^i)"^)^-^,!) ? ''^^ ^^-^ ^^^^ 

Dn+lUn = UnUn-lDn + (1 — an)In, 

for all n >0. 

Proof. First we consider the case that A, p are distinct partitions of n. Then 
in order to move from A to p by going up and then going down, one must 
add a box to a row of length A; of A and then remove a box from a row of 
length j. Similarly, in order to move from A to p by going down and then 
going up, one must remove a box from a row of length j of A, and then add 
a box to a row of length k. In both situations one has that j > 1, A; > and 
j ^ k -\- 1. From this it is straightforward to check (treating separately the 
cases that k > and k = 0), that the coefficient of p in 

(n + l)ie + n)DUn{X) - n{e + n - l)UDn{\) 

is 0. 

The second case to consider is that X = p are the same partition of n. 
Then j = /c + 1, and the coefficient of A in (n + 1){6 + n)DUn{X) is 

[9 + a/(A)][mi(A) + 1] + ^^[(A: - Q)mfc(A)]p + l){mk+i{\) + 1)]. 

k>l 

Similarly, the coefficient of A in n[9 + n — 1)C/Z)„(A) is 

mi(A)[0 + a(/(A) - 1)] + ^[(A: + l)m,+i(A)][(A: - a)(mfc(A) + 1)]. 

k>l 

Hence the coefficient of A in 

(n + l){9 + n)DUn{X) - n{e + n - l)UDn{\) 

is 

9 + al{X) + ami(A) + ^(A; - a)(A; + l)(mfc(A) - mfc+i(A)) 

k>l 

= 9 + al{\) - ami{X) + 2mi{X) 

+ rukiX) [{k + l)ik-a)-k{k-l- a)] 

k>2 

= 9 + al{X) - ami{X) + 2mi(A) + ^(2A: - a)mfc(A) 

k>2 

= 9 + 2n. 
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□ 

Corollary 17.21 determines the eigenvalues of the down- up walk on the King- 
man lattice with parameters 9, a. It is interesting that these are independent 
of the parameter a. We remark that since p{l) = p{0) = 1, the eigenvalue 
1 ~ n(e+n-i) Corollarv 17.21 has multiplicity 0. 

Corollary 7.2. Let p{j) denote the number of integer partitions ofj. Then 
the eigenvalues of UDn are 1 — ^Igiji^J"^ with multiplicity p{i) — p{i — 1) 
(0 < i < n). 

Proof. This is immediate from Theorem 14.11 and Proposition 17. 1[ □ 

Next we will study maximal separation distance for the down-up walk on 
the Kingman lattice. The first step is to determine where this is attained. 
Given a linear operator B : CP i— > CP, and partitions /i, A, it is convenient 
to let B[n, A] denote the coefficient of A in B{fi). 

Proposition 7.3. Let vr be the Pitman distribution with parameters 9 > 
and < a < 1. Let r be a non-negative integer. The quantity ^'^^(^j)'^'^ 
is minimized (among partitions n, X of size n) by ^ = {n),X = (1") or 
/i = (l"),A = (n). 

Proof. Proposition 14.51 gives that 

vr(A) vr(A) 

with all An{r, k) >0. The proposition now follows from three observations: 

• All terms in the sum are non-negative. Indeed, Proposition 14. 51 gives 
that all An{r, k) > 0, and U, D were defined probabilistically. 

• If /X = (n),A = (1") or fi = (1"),A = (n), then the summands for 
< k < n — 2 vanish. Indeed, for such k it is impossible to move 
from the partition fj, to the partition A by removing k boxes one at 
a time and then reattaching k boxes one at a time. 

• The k = n — 1 and k = n summands are each independent of both /i 
and A. Indeed, is equal to (1) for any partition ^ of size n. 
Since the up chain preserves the Pitman distribution, it follows that 
[/"""^[(l). A] = vr(A), so that the k = n — l summand is An{r,n — 1). 
Similarly, the k = n summand is An (r, n) . 

□ 

The following corollary will be helpful. 

Corollary 7.4. Consider down-up walk with parameters 9 > and < a < 
1 on the nth level of the Kingman lattice. Then s*{r) = ¥{T > r) where T is 
the sum of independent geometries with parameters ^{Q^n-i) 2 < i < n. 
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Proof. By Proposition [731 s*{r) = 1 - '^^^^l}^Jf^"''\ By Corollary [721 the 

down-up walk has n distinct eigenvalues, namely 1 and 1 — for 
2 < i < n. Since the distance between (n) and (1") is n — 1, the result 
follows from Propositions 12.3 1 and 12.41 □ 

Theorem 17.51 gives the precise asymptotic behavior of s*(r) in the special 
case that 9 = 1. 

Theorem 7.5. Let s*(r) be the maximal separation distance after r itera- 
tions of down-up walk on the Kingman lattice, in the special case that 6 = 1 
and < a < 1. 

(1) 

.*(r) = 2 f^i-^n^' - 1) , y^,, f 1 - 4 
(2) ForoO fixed, 

oo 

lim s*(cn2) = 2y(-l)^(i2 - 1)6"'=^'. 

n^oo ^— ' 

2 = 2 

Proof. By Proposition 17.31 one has that 

7r(l") 

By Corollary 17.21 the chain has n distinct eigenvalues. Since the distance 
between (n) and (1") is n — 1, it follows from Proposition 12.31 that 



s r 



^2 \ ^' 

'-^2) n 



n 

1=2 ^ ^ 2<j<n 

" / „-2 \ »■ „-2 

E n ' 

i=2 ^ ^ 2<j<n 



and the first assertion follows by elementary simplifications. 

For part 2 of the theorem, we claim that for c > fixed there is a constant 
ic (depending on c but not n) such that for i > ic, the summands in 

are decreasing in magnitude (and alternating in sign) . Part 2 of the theorem 
follows from this claim, since then one can take limits for each fixed i. To 
prove the claim, note that the summands are decreasing in magnitude if 
i > Vn, since one checks that (i^ — 1) is a decreasing function of 
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i when i > \/n. Since -, — , a, is a decreasing function of i, to handle 
i < \/n one needs only to show that 

- 1 „cn2[log(l-iV"'')-log(l-{i+l)V«')] > 1 



(i + 1)2-1 

for i > ic, a constant depending on c but not n. Using that log(l — x) > 
—X — x^ for < a; < ^ and that log(l — x) < —x for < x < 1, one has that 

cn^ [logfl - i^lr?) - log(l - (i + 1)V™^)1 > c(i + 1)^ - ci^ - > 2zc 

since i < y^. Clearly ^^_^^^y/_^ e2"^ > 1 for i large enough, completing the 
proof. □ 

For general values of 0, we do not have a result as precise as Theorem 
17.51 but obtain explicit upper and lower bounds for the separation distance 
mixing time. Note that when Q is fixed and n is growing, these bounds are 
of order . 

Corollary 7.6. Let n\ he the separation distance mixing time for down-up 

2 

walk (with parameters 6 > and < a < 1) on the nth level of Kingman's 
lattice. Then < ra* < 2E[T], where T is as in Corollary \7.4\ Moreover 

2 

if d = 1 then 



i=2 

and if 9 ^ 1 then 

n(^ + n-l), f{n + l){9 + l)\ ^ ^^rj.! _ ^ ri{e + n - 1 



/(n+l)(. + l)X ^ 
H 2(n + 0) ; - ^ J ^ 



.(^ + ^-1) 



n(9 + n — 1) , / n6 



1 v^ + ^- 

Proof Lemma Ogives that ^ < nt < 2E[r] and Corollary [73] gives that 

2 

'^[T] = J27=2 ^i'^e+i-i) • complete the proof of the upper bound, note that 
^ 1 1 ^ _ 1 1 f ^ 

^i(0 + i-i) "A W+t^ 0^ 

For the lower bound, note that 

1^ i(g + i-i) - A t{9 + t-l) * " °^ V 2(n + 0) 

□ 

To conclude, we relate separation distance of the up-down chain to sepa- 
ration distance of the down-up chain. 
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Proposition 7.7. Let s^j^ (r) be the maximal separation distance after r 
iterations of the down-up chain (with parameters 9 > and < a < 1) on 
the Kingman lattice, and let s*^fj^ (r) be the corresponding quantity for the 
up-down chain. Then 

shuA^) = stiD^+rir + 1) 

for all n > l,r > 0. 

Proof. The method is the same as for Proposition 15.71 The eigenvalues of 
DUn are 1 and := 1 - (^^^(g~^^^) (for 2 < j < n) yielding that 

j=2 k^j 

2<k<n 

The eigenvalues of UDn+i are 1 and tj (for 2 < j < n + 1) yielding that 

j=2 k^j ^ ^ '^^ 

2<k<n+l 

The result follows since tn+i = 0. □ 

8. Other examples 

This section treats other examples to which the commutation relation 
methodology applies. After discussing two classical examples (Bernoulli- 
Laplace models and subspace walks), we determine precise separation dis- 
tance asymptotics for a non-standard hypercube example. 

We focus on the down-up chain but for readers interested in the up-down 
chain mention the relation s*j^u^{r) = s^^^^_^{r + 1) (which is true for the 
same reasons as in the Young and Kingman examples). This holds for 
all examples in this section except for the subset walk on [^J sets or the 
subspace walk on [^J spaces (in these exceptional cases the two chains have 
the same separation distance asymptotics). 

8.1. Bernoulli-Laplace models. We analyze random walk on size j sub- 
sets of an n element set, where < 2j < n. Prom a subset S of size j, a 
step proceeds by first removing one of the j elements uniformly at random, 
and then randomly adding in one of the n — j + 1 elements in S — j. The 
stationary distribution is the uniform distribution on subsets of size j. This 
chain appears when analyzing the Bernoulli-Laplace model, in which there 
are two urns, the left containing j red balls, the right containing n — j black 
balls, and at each step a ball is picked uniformly at random in each urn, and 
the two balls are switched. 

It will be useful to let P be the Boolean lattice of rank n; the elements 
of P are the subsets of {1, • • • , n} and 5" < T in the partial order if 5 C T. 
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Letting U, D be the up and down operators for this poset, Stanley [5t3j 
observed that 

Dj+iUj = Uj-iDj + (n - 2j)Ij, 
for < j < n. For our purposes, it is more convenient to work with the 
normahzed operators 

Ui = -^Uj , Dj = -L>,-. 

IT' — J J 

Then the random walk on size j subsets of {1, • • • , n} is given by the operator 
UDj. Stanley's commutation relation becomes 

Dj+iUj = ajU^^iDj + (1 - aj)Ij 

^1^^ «J - U+l){n-j)- 

As a consequence of Theorem 14. H one obtains the eigenvalues of UDj. 
This goes back at least to Karlin and McGregor ^KMj . 

Corollary 8.1. The eigenvalues of UDj are 

{1 multiplicity 1 

1 - multvpUc^y (^) - (1 < . < 

Proposition 18.21 gives information about separation distance. The proof in 
[DFj used the theory of birth-death chains, and the fact that the Bernoulli- 
Laplace chain can be reduced to a birth death chain (look at the number of 
red balls in the right urn). Our proof uses commutation relations. 

Proposition 8.2. ( [DFj } Consider the random walk UDj on size j subsets 
of {!,••• ,n}. Let r be a non-negative integer, and let vr be the uniform 
distribution on j element subsets of {!,■ ■ ■ ,n}. 

(1) The quantity is minimized (among pairs of j element sub- 

sets of {!,■ • • ,n}) by any S, T such that 5 n T = 0. 

(2) 

s*(r) = F{X > r), 
where X is the sum of independent geometries having parameters 

i{n-i+l) r 

Proof. Given a linear operator A : CP i— > CP, and subsets S", T of {1, • • • ,n} 
of size j, let A[S, T] denote the coefficient of T in A{S). Proposition [13] gives 
that 

{UDnS,T] &^D^\S^ 
AT) vr(r) 

with all Aj{r,k) > 0. The first part of the proposition now follows from 
three observations: 

• All terms in the sum are non-negative. Indeed, all Aj(r, k) > and 
U, D were defined probabilistically. 



COMMUTATION RELATIONS AND MARKOV CHAINS 



31 



• If S n T = 0, then the summands for < A; < j — 1 ah vanish. This 
is clear since for such k, D^[S,T] = 0. 

• The k = j summand is independent of both S and T. Indeed, 
D^{S) = for any S of size j, and 11^(9) is uniformly distributed 
among the size j subsets of {!,••• ,n}. Hence the k = j summand 
is equal to Aj{r,j). 

For the second part of the proposition, Corollary 18.11 gives that UDj has 
j + 1 distinct eigenvalues. Letting x = S,y = T where 5* H T = 0, one 
has that dist{x,y) = j. The result now follows from Propositions 12.3] and 
[231 □ 



In fact there is another proof of part 2 of Proposition 18.21 which uses only 
combinatorial properties of the sequence Aj(r,j). 

Proof. (Second proof of part 2 of Proposition 18. 2p The proof of part 1 of 
Proposition 18.21 gives that s*{r) = 1 — Aj{r,j), where Aj{r,j) is defined in 
Proposition 14.51 Letting denote the coefficient of x" in a power 

series f{x), Proposition 14.61 gives that 



„j rr-?' 0-»+i)("-i+») 

I ^^»=l jin-j+l) 

lli=l^ j{n-j+l) 

j{n-j+l) 



=1 



j(n-j+l) 



j i(n-i+l) 



l-X^^ i{n-i+l) 



i=l 



j{n-j+l) 



The last step used the change of variables i j + 1 — i. 

Note that if Z is geometric with parameter p, then Z has probability 
generating function 



xp 



1 — x(l — p) 



Thus Aj{r,j) is the probability that the convolution of geometries with 

i(ra— i+l) 
j(n-j+l) 



parameters is at most r, and the result follows. □ 



The asymptotic behavior of s*(r) (in continuous time) is studied in detail 
in [DSaj . using a continuous time analog of part 2 of Proposition 18.21 (in 
which geometries are replaced by exponentials). A similar analysis can be 
carried out in discrete time. For instance if j < ^ tends to infinity, there is 

a separation cutoff at time tnj = log(j). For information concerning 

convergence in the total variation metric, see [Be] or [DSh| . 
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8.2. Subspace walks. This is a g'-analog of the previous example. The 
random walk is on j-dimensional subspaces of an n-dimensional vector space 
over a finite field F^, where < 2j < n. From a j-dimensional subspace S, 
a step of the walk proceeds by first choosing uniformly at random a j — 1 
dimensional subspace W contained in S, and then choosing uniformly at 
random a j dimensional subspace T containing W. 

Up to holding, this random walk is equivalent to the nearest neighbor walk 
on the graph of j dimensional subspaces, where two subspaces are connected 
by an edge if their intersection has dimension j — 1. As discussed in |Be| . 
|D'A| . the eigenvalues of this walk are known and sharp total variation 
distance estimates can be obtained by studying a related birth-death chain 
on {0, • • • ,j}, which is just the associated graph distance process. 

To revisit this example using commutation relations, let P be the subspace 
lattice of an n-dimensional vector space over a finite field ¥q. Letting U, D 
be the up and down operators for the poset P, Stanley \St3\ observed that 

for < j < n. For our purposes it is convenient to renormalize the operators 
as 

J qn-j _ 1 J ' J qJ -1 ^ 

Then the random walk on j dimensional subspaces is given by UDj, and 
one checks that the commutation relation becomes 

Dj+iUj = ajUj^iDj + (1 - aj)Ij 

^^ere aj = [^L't^jfai+ilij ■ 

As an immediate consequence of this commutation relation and Theorem 
14.11 one obtains the eigenvalues of the subspace walk. 

Corollary 8.3. The eigenvalues of UDj are 

{1 multiplicity 1 

1 - rnultzpUcUy [^] - [ T^] (1 < ^ < j) 

Here ["] denotes the number of i- dimensional subspaces of an n-dimensional 
vector space over F^. 

Proposition 18.41 gives a result about separation distance. This also follows 
from the birth-death chain theory in |DFj . 

Proposition 8.4. Consider the random walk UDj on j-dimensional sub- 
spaces of an n dimensional vector space V over Fg. Let r be a non-negative 
integer, and let vr be the uniform distribution on j-dimensional subspaces of 
V. 

(1) The quantity is minimized (among pairs of j dimensional 

subspaces of V) by any S, T such that 5 n T = 0. 
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(2) One has that s*{r) = ¥{X > r), where X is the sum of independent 
geometries with parameters , for I < i < j. 

Proof. The proof method for both parts is the same as for the proof of 
Proposition I8.2| one need only replace the word "subset" by "subspace" 
and the word "size" by "dimension" . Note that the second proof of part of 
Proposition 18.21 also carries over to the subspace setting. □ 



Concerning the asymptotic behavior of s*{r), we note that [DSaj gives 
results (in the continuous time case) , using an analog of part 2 of Proposition 
18.41 in which the geometries are replaced by exponentials. Their method can 
be transferred to the discrete time setting. For instance if i < f tends to 
infinity, there is a separation cutoff at time tnj = j- 

8.3. Gibbs sampler for hypercube. The main object of study in this 
example is the birth-death chain on the set {0, ,n} with transition 
probabilities 

K{x, x-l) = -{l- p), K{x, x) = -p+(l--\{l-p) 
n n \ nJ 

K{x,x + I) =p(^-^ 

Here < p < 1 and the stationary distribution of this chain is the p-binomial 
distribution 7r(x) = (^)p^(l — pY~^ . 

We remark that this Markov chain is the distance chain for the Gibbs 
sampler on the hypercube, used to sample from the distribution in which a 
length n 0-1 vector is assigned probability — where x is the num- 

ber of I's in the vector. For general p we have not seen this exact analyzed 
chain in the literature (though possibly it has been studied). Different birth- 
death chains with the same stationary distribution are studied as examples 
in [DSaj. Our birth-death chain has the property that the eigenvalues are 
independent of p (see Corollarv l8.6p : the examples in [DSa] do not. 

To motivate the definition of up and down operators, we note that the 
birth-death chain in this section is, in the terminology of Section [3l an 
example of a down-up Markov chain. The poset we use is Pascal's lattice: 
the vertices of the nth level are labeled by pairs {x, n) where x = 0, 1, • • • , n. 
The only edges are (x, n) Z' (x, n + 1) and (x, n) /" (x + 1, n + 1), each with 
multiplicity 1. Then the dimension of the vertex (x,n) is ("). One checks 
that the probability distributions M„((x,n)) = (")p^(l — p)""^ are coherent 
with respect to Pascal's lattice [K], and computes that the corresponding 
up and down chains are given by 

[/„[(x, n)] = (1 - p) • (x, n + 1) + p • (x + 1, n + 1) 

D„[(x, n)] = ("l - - V (x, n - 1) + - • (x - 1, n - 1). 
V n/ n 

From this one sees that our birth-death chain is precisely the down-up chain 

VDr, on Pascal's lattice. 
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Proposition 8.5. Letting an = one has that 

Dn+lUn = a-nUn-lDn + (1 — an)In- 

Proof. This is straightforward to check from the definitions of U and D. □ 

Corollary 18.61 determines the eigenvalues of the down-up walk on Pascal's 
lattice. It is curious that they are independent of p. 

Corollary 8.6. The eigenvalues of UDn are 1 — ^ with multiplicity 1, for 
0<i<n. 

Proof. This is immediate from Theorem 14.11 and Proposition 18. 5i □ 

Proposition 18.71 determines where the maximal separation distance is at- 
tained. 

Proposition 8.7. Let vr be the p-hinomial distribution and let r be a non- 
negative integer. The quantity is minimized (among < 
x,y < n) by X = 0,y = n or x = n,y = 0. 

Proof. Given a linear operator B : CPn i— > CP„, let B[{x,n), (y,n)] denote 
the coefficient of {y,n) in B(x,n). Proposition 14.51 gives that 

iUDY[ix,n),{y,n)] _^ ^ n), (j/, n)] 

7 ^ ~ Z^^nV, K) - - , 

T^iy^n) ^ vr(y,n) 

with all An{r, k) > 0. The proposition now follows from three facts: 

• All terms in the sum are non- negative. Indeed, all An{r, k) > and 
U, D were defined probabilistically. 

• If X = 0, y = n or x = n, y = 0, the summands for < A; < n — 1 all 
vanish. 

• The k = n summand is independent of both x and y. Indeed, 
D^{x,n) = (0,0) and the coefficient of in C/"(0,0) is n^y^n). 
So the k = n summand is exactly An{r,n). 

□ 

Finally, we determine the exact asymptotic behavior of s*{r) for this 
example. 

Proposition 8.8. Consider the random walk UDn corresponding to the p- 
binomial distribution. Let r be a non-negative integer. 

(1) s*{r) = ¥{X > r) where X is the sum of independent geometries 
with parameters ^ for 1 < i < n. 

(2) s*{r) = 1 — "'"^^^'"^ where S{r,k) is a Stirling number of the second 
kind (i.e. the number of partitions of an r set into k blocks). 

(3) For c fixed in M and n — > oo, 

s* (n log(n) + cn) = 1 — e~^ + O 
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Proof. Proposition EZl gives that s*(r) = 1 - ^^^^"^^"'^"^^'"^^ . By Corollary 
I8.6l the chain has n + 1 distinct eigenvalues. Hence the first assertion follows 
from Proposition 12.31 (with x = (0,n) and y = (n, n)), and Proposition 12.41 
For the second assertion, it follows from the proof of Proposition 18.71 
and Proposition 14.51 that s*{r) = 1 — A„(r, n) where An{r,k) satisfies the 
recurrence 

Anir, k) = '^~^^'^ An{r - 1, /c - 1) + -An{r - 1, k) 
n n 

with initial condition 74„(0, m) = (^o,m- It is straightforward to check that 

An{r,k) = : 

n^fn-'ky. solves the recurrence, using the recurrence for Stirling 

numbers 

5(r, k) = 5(r - 1, A; - 1) + kS{r - 1, A;) 

on page 33 of |Stl] . 

For the third assertion, it follows from the second assertion and the argu- 
ment in part 2 of Theorem 15 . 51 that s*(r) = 1 — P(n, r, n), where P(n, r, n) is 
the probability of n occupied boxes when r balls are dropped into n boxes. 
The result now follows from asymptotics of the coupon collector's problem, 
as in the proof of Theorem 15.51 □ 

Remark: The waiting time for n boxes to all be occupied when balls are 
randomly dropped into them one at a time is a convolution of independent 
geometries with parameters ^ for \ < i < n. Thus part 3 of Proposition 18.81 
can be proved without using part 2 of Proposition 18. 8[ Our reason for using 
part 2 was to illustrate that one can sometimes usefully solve the recursion 
for the combinatorially defined quantities An (r, k) . 
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